Ideas and exercises come from https://r4ds.had.co.nz/transform.html
Additional notes by TCS
Setup
First, we load the tidyverse package and a dataset. This
data frame contains all 336,776 flights that departed from New York City
in 2013.
require(nycflights13)
Loading required package: nycflights13
require(tidyverse)
Loading required package: tidyverse
Registered S3 methods overwritten by 'dbplyr':
method from
print.tbl_lazy
print.tbl_sql
── Attaching packages ──────── tidyverse 1.3.2 ──✔ ggplot2 3.3.6 ✔ purrr 0.3.4
✔ tibble 3.1.8 ✔ dplyr 1.0.10
✔ tidyr 1.2.1 ✔ stringr 1.4.1
✔ readr 2.1.2 ✔ forcats 0.5.2 ── Conflicts ─────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
flights
Basic dplyr functions
There are 5 key functions (“verbs”), plus a helper function, that do
most data manipulation tasks in dplyr:
filter: pick observations by values
arrange: reorder rows
select: pick variables by name
mutate: create new variables from existing ones, using
functions
summarise: collapse values into single ones
group_by: change scope of a verb from the whole dataset
to individual groups
How verbs work
Input and output are data frames. The input is never modified.
Arguments consist of
1. A data frame
2. “What to do with the data frame”
filter() gives you a subset of rows based on
values
Available comparison operators are >, >=, <, <=, != (not
equal), and == (equal).
Wrapping the assignment in parentheses also prints out a preview of
the resulting dataframe.
jan1 <- filter(flights, month == 1, day == 1)
jan1
(dec25 <- filter(flights, month == 12, day == 25))
For floating-point numbers, instead of relying on ==, use near() to
avoid unwanted inequality due to rounding:
sqrt(2) ^ 2 == 2
[1] FALSE
1 / 49 * 49 == 1
[1] FALSE
near(sqrt(2) ^ 2, 2)
[1] TRUE
near(1 / 49 * 49, 1)
[1] TRUE
You can also use:
* logical operators &, |, !, xor
* %in% constructions
(nov_dec <- filter(flights, month == 11 | month == 12))
(jan_mar <- filter(flights, month %in% seq(1,3)))
NA handling in filter
filter() includes ONLY rows where the condition is TRUE;
it excludes both FALSE and NA values. If you want to preserve missing
values, ask for them explicitly:
df <- tibble(x = c(1, NA, 3))
(biggerthan1 <- filter(df, x > 1))
(bigorNA <- filter(df, is.na(x) | x > 1))
Exercises for filter()
- Find all flights that
- Had an arrival delay of two or more hours
- Flew to Houston (IAH or HOU)
- Were operated by United, American, or Delta
- Departed in summer (July, August, and September)
- Arrived more than two hours late, but didn’t leave late
- Were delayed by at least an hour, but made up over 30 minutes in
flight
- Departed between midnight and 6am (inclusive)
- Another useful dplyr filtering helper is
between().
What does it do? Can you use it to simplify the code needed to answer
the previous challenges?
- How many flights have a missing dep_time? What other variables are
missing? What might these rows represent?
- Why is NA ^ 0 not missing? Why is NA | TRUE not missing? Why is
FALSE & NA not missing? Can you figure out the general rule? (NA * 0
is a tricky counterexample!)
# from ?flights we learn that the delay columns are in minutes
(delay2hr <- filter(flights, arr_delay>=120))
(intohouston <- filter(flights, dest == "IAH" | dest == "HOU"))
(someairlines <- filter(flights, carrier %in% c("UA", "AA", "DL")))
(summer <- filter(flights, month %in% c(7,8,9)))
(gotlate <- filter(flights, dep_delay <= 0 & arr_delay >= 120))
(madeup <- filter(flights, dep_delay >= 60 & (dep_delay-arr_delay > 30)))
(redeye <- filter(flights, dep_time < 600 | dep_time == 2400)) #2400 is midnight
(betweensummer <- filter(flights, between(month, 7, 9))) # inclusive
(missingdeptime <- filter(flights, is.na(dep_time) == TRUE)) # all missing arrival times and some missing tail numbers. probably cancelled flights
(NA^0)
[1] 1
(NA|TRUE)
[1] TRUE
(FALSE & NA)
[1] FALSE
(NA * 0)
[1] NA
(Inf * 0)
[1] NaN
(Inf ^ 1)
[1] Inf
Notes on NAs
A missing value can look like a real one. Certain operations always
give a numerical or logical result:
* NA ^ 0 = 1
* NA|TRUE = TRUE because one of the arguments is true
* FALSE & NA = FALSE because one of the arguments is false
NA * 0 is “a tricky counterexample”. You would think that anything
times 0 is 0. However, it could be infinity (Inf) which would be
undefined. Hence the expression cannot be evaluated. (Unlike Inf^0 which
still = 1.)
arrange() is for sorting rows
- The arguments for
arrange() are a dataframe and column
name(s).
- The
desc option reverses the order.
- Missing values (NAs) are always at the end regardless of
desc()
Remember that rows are in no particular order even if you’ve appended
them in some order!
(bydate <- arrange(flights, year, month, day))
(longestfirst <- arrange(flights, desc(dep_delay)))
df <- tibble(x = c(5, 2, NA))
(mytib <- arrange(df, x))
(mytibdesc <- arrange(df, desc(x)))
Exercises for arrange()
- How could you use arrange() to sort all missing values to the start?
(Hint: use is.na()).
Note: FALSE (0) sorts before TRUE (1) so we need to use
!is.na()
- Sort flights to find the most delayed flights. Find the flights that
left earliest.
- Sort flights to find the fastest (highest speed) flights.
- Which flights travelled the farthest? Which travelled the
shortest?
(nafirst <- arrange(flights, !is.na(dep_time))) # FALSE comes before TRUE
(earliestdelayed <- arrange(flights, desc(dep_delay), dep_time))
(fastest <- arrange(flights, desc(distance/air_time)))
(farthest <- arrange(flights, desc(distance)))
(shortestflights <- arrange(flights, distance))
select() gives you a subset of columns by name
You can name each column, or specify a range using a colon.
As with other R selections, you can omit certain columns using the minus
sign.
(datedata <- select(flights, year, month, day))
(bunchocols <- select(flights, year:day))
(nodates <- select(flights, -(year:day)))
Partial names
select() does not have to use exact column matches.
You can use partial names and regular expressions:
starts_with("foo")
ends_with("bar")
contains("foobar")
matches(some_regex)
num_range("x", 1:3) matches x1, x2 and x3
Variants on select()
You can use select() to rename and re-organize columns to some
extent. For example:
rename() is considered a variant of select() where you
take a column, change its name, and keep all other columns as well. If
you use select() to rename a column you will lose all other
columns.
everything() is a helper for select() that lets you
move one or a few columns to the beginning (left) of the table, while
retaining all other columns.
Exercises for select()
Brainstorm as many ways as possible to select dep_time,
dep_delay, arr_time, and arr_delay from flights.
What happens if you include the name of a variable multiple times
in a select() call?
What does the any_of() function do? Why might it be helpful in
conjunction with this vector?
vars <- c("year", "month", "day", "dep_delay", "arr_delay")
- Does the result of running the following code surprise you? How do
the select helpers deal with case by default? How can you change that
default?
select(flights, contains("TIME"))
LS0tCnRpdGxlOiAiTm90ZXMgb24gUiBmb3IgRGF0YSBTY2llbmNlIENoYXB0ZXIgNTogRGF0YSB0cmFuc2Zvcm1hdGlvbiIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKKklkZWFzIGFuZCBleGVyY2lzZXMgY29tZSBmcm9tIGh0dHBzOi8vcjRkcy5oYWQuY28ubnovdHJhbnNmb3JtLmh0bWwqCgoqQWRkaXRpb25hbCBub3RlcyBieSBUQ1MqCgojIFNldHVwCkZpcnN0LCB3ZSBsb2FkIHRoZSBgdGlkeXZlcnNlYCBwYWNrYWdlIGFuZCBhIGRhdGFzZXQuIFRoaXMgZGF0YSBmcmFtZSBjb250YWlucyBhbGwgMzM2LDc3NiBmbGlnaHRzIHRoYXQgZGVwYXJ0ZWQgZnJvbSBOZXcgWW9yayBDaXR5IGluIDIwMTMuCgpgYGB7ciBzZXR1cH0KcmVxdWlyZShueWNmbGlnaHRzMTMpCnJlcXVpcmUodGlkeXZlcnNlKQpmbGlnaHRzCmBgYAoKIyBCYXNpYyBkcGx5ciBmdW5jdGlvbnMKClRoZXJlIGFyZSA1IGtleSBmdW5jdGlvbnMgKCJ2ZXJicyIpLCBwbHVzIGEgaGVscGVyIGZ1bmN0aW9uLCB0aGF0IGRvIG1vc3QgZGF0YSBtYW5pcHVsYXRpb24gdGFza3MgaW4gZHBseXI6ICAKCiogYGZpbHRlcmA6IHBpY2sgb2JzZXJ2YXRpb25zIGJ5IHZhbHVlcwoqIGBhcnJhbmdlYDogcmVvcmRlciByb3dzICAKKiBgc2VsZWN0YDogcGljayB2YXJpYWJsZXMgYnkgbmFtZSAgCiogYG11dGF0ZWA6IGNyZWF0ZSBuZXcgdmFyaWFibGVzIGZyb20gZXhpc3Rpbmcgb25lcywgdXNpbmcgZnVuY3Rpb25zICAKKiBgc3VtbWFyaXNlYDogY29sbGFwc2UgdmFsdWVzIGludG8gc2luZ2xlIG9uZXMgIAoqIGBncm91cF9ieWA6IGNoYW5nZSBzY29wZSBvZiBhIHZlcmIgZnJvbSB0aGUgd2hvbGUgZGF0YXNldCB0byBpbmRpdmlkdWFsIGdyb3VwcyAKCiMjIEhvdyB2ZXJicyB3b3JrCgpJbnB1dCBhbmQgb3V0cHV0IGFyZSBkYXRhIGZyYW1lcy4gVGhlIGlucHV0IGlzIG5ldmVyIG1vZGlmaWVkLiAgICAKCkFyZ3VtZW50cyBjb25zaXN0IG9mICAKMS4gQSBkYXRhIGZyYW1lICAKMi4gIldoYXQgdG8gZG8gd2l0aCB0aGUgZGF0YSBmcmFtZSIgICAKCiMjIGBmaWx0ZXIoKWAgZ2l2ZXMgeW91IGEgc3Vic2V0IG9mIHJvd3MgYmFzZWQgb24gdmFsdWVzCgpBdmFpbGFibGUgY29tcGFyaXNvbiBvcGVyYXRvcnMgYXJlICA+LCA+PSwgPCwgPD0sICE9IChub3QgZXF1YWwpLCBhbmQgPT0gKGVxdWFsKS4KCgpXcmFwcGluZyB0aGUgYXNzaWdubWVudCBpbiBwYXJlbnRoZXNlcyBhbHNvIHByaW50cyBvdXQgYSBwcmV2aWV3IG9mIHRoZSByZXN1bHRpbmcgZGF0YWZyYW1lLgoKCgpgYGB7ciBmaWx0ZXIgZXhhbXBsZXN9CmphbjEgPC0gZmlsdGVyKGZsaWdodHMsIG1vbnRoID09IDEsIGRheSA9PSAxKQpqYW4xCgooZGVjMjUgPC0gZmlsdGVyKGZsaWdodHMsIG1vbnRoID09IDEyLCBkYXkgPT0gMjUpKQpgYGAKCkZvciBmbG9hdGluZy1wb2ludCBudW1iZXJzLCBpbnN0ZWFkIG9mIHJlbHlpbmcgb24gPT0sIHVzZSBuZWFyKCkgdG8gYXZvaWQgdW53YW50ZWQgaW5lcXVhbGl0eSBkdWUgdG8gcm91bmRpbmc6CgpgYGB7ciBuZWFyIGV4YW1wbGUsIGVjaG8gPSBUUlVFfQpzcXJ0KDIpIF4gMiA9PSAyCjEgLyA0OSAqIDQ5ID09IDEKbmVhcihzcXJ0KDIpIF4gMiwgIDIpCm5lYXIoMSAvIDQ5ICogNDksIDEpCmBgYAoKWW91IGNhbiBhbHNvIHVzZTogIAoqIGxvZ2ljYWwgb3BlcmF0b3JzICYsIHwsICEsIHhvciAgCiogYCVpbiVgIGNvbnN0cnVjdGlvbnMgIAoKYGBge3IgbG9naWNhbCBmaWx0ZXIgZXhhbXBsZXN9Cgoobm92X2RlYyA8LSBmaWx0ZXIoZmxpZ2h0cywgbW9udGggPT0gMTEgfCBtb250aCA9PSAxMikpCgooamFuX21hciA8LSBmaWx0ZXIoZmxpZ2h0cywgbW9udGggJWluJSBzZXEoMSwzKSkpCmBgYAoKIyMgTkEgaGFuZGxpbmcgaW4gYGZpbHRlcmAKCmBmaWx0ZXIoKWAgaW5jbHVkZXMgT05MWSByb3dzIHdoZXJlIHRoZSBjb25kaXRpb24gaXMgVFJVRTsgaXQgZXhjbHVkZXMgYm90aCBGQUxTRSBhbmQgTkEgdmFsdWVzLiBJZiB5b3Ugd2FudCB0byBwcmVzZXJ2ZSBtaXNzaW5nIHZhbHVlcywgYXNrIGZvciB0aGVtIGV4cGxpY2l0bHk6ICAKCmBgYHtyIGZpbHRlciBOQSBleGFtcGxlcywgZWNobyA9IFRSVUV9CmRmIDwtIHRpYmJsZSh4ID0gYygxLCBOQSwgMykpCihiaWdnZXJ0aGFuMSA8LSBmaWx0ZXIoZGYsIHggPiAxKSkKKGJpZ29yTkEgPC0gZmlsdGVyKGRmLCBpcy5uYSh4KSB8IHggPiAxKSkKYGBgCgojIyBFeGVyY2lzZXMgZm9yIGBmaWx0ZXIoKWAKCjEuIEZpbmQgYWxsIGZsaWdodHMgdGhhdCAgCisgSGFkIGFuIGFycml2YWwgZGVsYXkgb2YgdHdvIG9yIG1vcmUgaG91cnMKKyBGbGV3IHRvIEhvdXN0b24gKElBSCBvciBIT1UpICAKKyBXZXJlIG9wZXJhdGVkIGJ5IFVuaXRlZCwgQW1lcmljYW4sIG9yIERlbHRhICAKKyBEZXBhcnRlZCBpbiBzdW1tZXIgKEp1bHksIEF1Z3VzdCwgYW5kIFNlcHRlbWJlcikgIAorIEFycml2ZWQgbW9yZSB0aGFuIHR3byBob3VycyBsYXRlLCBidXQgZGlkbuKAmXQgbGVhdmUgbGF0ZSAgCisgV2VyZSBkZWxheWVkIGJ5IGF0IGxlYXN0IGFuIGhvdXIsIGJ1dCBtYWRlIHVwIG92ZXIgMzAgbWludXRlcyBpbiBmbGlnaHQgIAorIERlcGFydGVkIGJldHdlZW4gbWlkbmlnaHQgYW5kIDZhbSAoaW5jbHVzaXZlKSAgCjIuIEFub3RoZXIgdXNlZnVsIGRwbHlyIGZpbHRlcmluZyBoZWxwZXIgaXMgYGJldHdlZW4oKWAuIFdoYXQgZG9lcyBpdCBkbz8gQ2FuIHlvdSB1c2UgaXQgdG8gc2ltcGxpZnkgdGhlIGNvZGUgbmVlZGVkIHRvIGFuc3dlciB0aGUgcHJldmlvdXMgY2hhbGxlbmdlcz8gIAozLiBIb3cgbWFueSBmbGlnaHRzIGhhdmUgYSBtaXNzaW5nIGRlcF90aW1lPyBXaGF0IG90aGVyIHZhcmlhYmxlcyBhcmUgbWlzc2luZz8gV2hhdCBtaWdodCB0aGVzZSByb3dzIHJlcHJlc2VudD8gIAo0LiBXaHkgaXMgTkEgXiAwIG5vdCBtaXNzaW5nPyBXaHkgaXMgTkEgfCBUUlVFIG5vdCBtaXNzaW5nPyBXaHkgaXMgRkFMU0UgJiBOQSBub3QgbWlzc2luZz8gQ2FuIHlvdSBmaWd1cmUgb3V0IHRoZSBnZW5lcmFsIHJ1bGU/IChOQSAqIDAgaXMgYSB0cmlja3kgY291bnRlcmV4YW1wbGUhKSAgCgpgYGB7ciBmaWx0ZXIgZXhlcmNpc2VzLCBlY2hvID0gVFJVRX0KIyBmcm9tID9mbGlnaHRzIHdlIGxlYXJuIHRoYXQgdGhlIGRlbGF5IGNvbHVtbnMgYXJlIGluIG1pbnV0ZXMKKGRlbGF5MmhyIDwtIGZpbHRlcihmbGlnaHRzLCBhcnJfZGVsYXk+PTEyMCkpCihpbnRvaG91c3RvbiA8LSBmaWx0ZXIoZmxpZ2h0cywgZGVzdCA9PSAiSUFIIiB8IGRlc3QgPT0gIkhPVSIpKQooc29tZWFpcmxpbmVzIDwtIGZpbHRlcihmbGlnaHRzLCBjYXJyaWVyICVpbiUgYygiVUEiLCAiQUEiLCAiREwiKSkpCihzdW1tZXIgPC0gZmlsdGVyKGZsaWdodHMsIG1vbnRoICVpbiUgYyg3LDgsOSkpKQooZ290bGF0ZSA8LSBmaWx0ZXIoZmxpZ2h0cywgZGVwX2RlbGF5IDw9IDAgJiBhcnJfZGVsYXkgPj0gMTIwKSkKKG1hZGV1cCA8LSBmaWx0ZXIoZmxpZ2h0cywgZGVwX2RlbGF5ID49IDYwICYgKGRlcF9kZWxheS1hcnJfZGVsYXkgPiAzMCkpKQoocmVkZXllIDwtIGZpbHRlcihmbGlnaHRzLCBkZXBfdGltZSA8IDYwMCB8IGRlcF90aW1lID09IDI0MDApKSAjMjQwMCBpcyBtaWRuaWdodAooYmV0d2VlbnN1bW1lciA8LSBmaWx0ZXIoZmxpZ2h0cywgYmV0d2Vlbihtb250aCwgNywgOSkpKSAjIGluY2x1c2l2ZQoobWlzc2luZ2RlcHRpbWUgPC0gZmlsdGVyKGZsaWdodHMsIGlzLm5hKGRlcF90aW1lKSA9PSBUUlVFKSkgIyBhbGwgbWlzc2luZyBhcnJpdmFsIHRpbWVzIGFuZCBzb21lIG1pc3NpbmcgdGFpbCBudW1iZXJzLiBwcm9iYWJseSBjYW5jZWxsZWQgZmxpZ2h0cwpgYGAKYGBge3IgbW9yZSBOQSBleGFtcGxlcywgZWNobz1UUlVFfQooTkFeMCkKKE5BfFRSVUUpCihGQUxTRSAmIE5BKQooTkEgKiAwKQooSW5mICogMCkKKEluZiBeIDEpCmBgYAojIyBOb3RlcyBvbiBOQXMgIApBIG1pc3NpbmcgdmFsdWUgY2FuIGxvb2sgbGlrZSBhIHJlYWwgb25lLiBDZXJ0YWluIG9wZXJhdGlvbnMgYWx3YXlzIGdpdmUgYSBudW1lcmljYWwgb3IgbG9naWNhbCByZXN1bHQ6ICAgICAgCiogTkEgXiAwID0gMSAgCiogTkF8VFJVRSA9IFRSVUUgYmVjYXVzZSBvbmUgb2YgdGhlIGFyZ3VtZW50cyBpcyB0cnVlICAKKiBGQUxTRSAmIE5BID0gRkFMU0UgYmVjYXVzZSBvbmUgb2YgdGhlIGFyZ3VtZW50cyBpcyBmYWxzZSAgCgpOQSAqIDAgaXMgImEgdHJpY2t5IGNvdW50ZXJleGFtcGxlIi4gWW91IHdvdWxkIHRoaW5rIHRoYXQgYW55dGhpbmcgdGltZXMgMCBpcyAwLiBIb3dldmVyLCBpdCBjb3VsZCBiZSBpbmZpbml0eSAoSW5mKSB3aGljaCB3b3VsZCBiZSB1bmRlZmluZWQuIEhlbmNlIHRoZSBleHByZXNzaW9uIGNhbm5vdCBiZSBldmFsdWF0ZWQuIChVbmxpa2UgSW5mXjAgd2hpY2ggc3RpbGwgPSAxLikgIAoKIyBgYXJyYW5nZSgpYCBpcyBmb3Igc29ydGluZyByb3dzCgoqIFRoZSBhcmd1bWVudHMgZm9yIGBhcnJhbmdlKClgIGFyZSBhIGRhdGFmcmFtZSBhbmQgY29sdW1uIG5hbWUocykuICAgCiogVGhlIGBkZXNjYCBvcHRpb24gcmV2ZXJzZXMgdGhlIG9yZGVyLiAgCiogTWlzc2luZyB2YWx1ZXMgKE5BcykgYXJlIGFsd2F5cyBhdCB0aGUgZW5kIHJlZ2FyZGxlc3Mgb2YgYGRlc2MoKWAgCgpSZW1lbWJlciB0aGF0IHJvd3MgYXJlIGluIG5vIHBhcnRpY3VsYXIgb3JkZXIgZXZlbiBpZiB5b3UndmUgYXBwZW5kZWQgdGhlbSBpbiBzb21lIG9yZGVyISAgCgpgYGB7ciBhcnJhbmdlIGV4YW1wbGVzLCBlY2hvID0gVFJVRX0KKGJ5ZGF0ZSA8LSBhcnJhbmdlKGZsaWdodHMsIHllYXIsIG1vbnRoLCBkYXkpKQoobG9uZ2VzdGZpcnN0IDwtIGFycmFuZ2UoZmxpZ2h0cywgZGVzYyhkZXBfZGVsYXkpKSkKZGYgPC0gdGliYmxlKHggPSBjKDUsIDIsIE5BKSkKKG15dGliIDwtIGFycmFuZ2UoZGYsIHgpKQoobXl0aWJkZXNjIDwtIGFycmFuZ2UoZGYsIGRlc2MoeCkpKQpgYGAKCiMjIEV4ZXJjaXNlcyBmb3IgYGFycmFuZ2UoKWAgIAoxLiBIb3cgY291bGQgeW91IHVzZSBhcnJhbmdlKCkgdG8gc29ydCBhbGwgbWlzc2luZyB2YWx1ZXMgdG8gdGhlIHN0YXJ0PyAoSGludDogdXNlIGlzLm5hKCkpLiAgCgoqTm90ZToqIEZBTFNFICgwKSBzb3J0cyBiZWZvcmUgVFJVRSAoMSkgc28gd2UgbmVlZCB0byB1c2UgIWlzLm5hKCkgIAoKMi4gU29ydCBmbGlnaHRzIHRvIGZpbmQgdGhlIG1vc3QgZGVsYXllZCBmbGlnaHRzLiBGaW5kIHRoZSBmbGlnaHRzIHRoYXQgbGVmdCBlYXJsaWVzdC4gIAozLiBTb3J0IGZsaWdodHMgdG8gZmluZCB0aGUgZmFzdGVzdCAoaGlnaGVzdCBzcGVlZCkgZmxpZ2h0cy4gIAo0LiBXaGljaCBmbGlnaHRzIHRyYXZlbGxlZCB0aGUgZmFydGhlc3Q/IFdoaWNoIHRyYXZlbGxlZCB0aGUgc2hvcnRlc3Q/ICAKCmBgYHtyIGFycmFuZ2UgZXhlcmNpc2VzLCBlY2hvID0gVFJVRX0KCihuYWZpcnN0IDwtIGFycmFuZ2UoZmxpZ2h0cywgIWlzLm5hKGRlcF90aW1lKSkpICMgRkFMU0UgY29tZXMgYmVmb3JlIFRSVUUKKGVhcmxpZXN0ZGVsYXllZCA8LSBhcnJhbmdlKGZsaWdodHMsIGRlc2MoZGVwX2RlbGF5KSwgZGVwX3RpbWUpKQooZmFzdGVzdCA8LSBhcnJhbmdlKGZsaWdodHMsIGRlc2MoZGlzdGFuY2UvYWlyX3RpbWUpKSkKKGZhcnRoZXN0IDwtIGFycmFuZ2UoZmxpZ2h0cywgZGVzYyhkaXN0YW5jZSkpKQooc2hvcnRlc3RmbGlnaHRzIDwtIGFycmFuZ2UoZmxpZ2h0cywgZGlzdGFuY2UpKQpgYGAKIyBgc2VsZWN0KClgIGdpdmVzIHlvdSBhIHN1YnNldCBvZiBjb2x1bW5zIGJ5IG5hbWUKCllvdSBjYW4gbmFtZSBlYWNoIGNvbHVtbiwgb3Igc3BlY2lmeSBhIHJhbmdlIHVzaW5nIGEgY29sb24uICAKQXMgd2l0aCBvdGhlciBSIHNlbGVjdGlvbnMsIHlvdSBjYW4gb21pdCBjZXJ0YWluIGNvbHVtbnMgdXNpbmcgdGhlIG1pbnVzIHNpZ24uCgpgYGB7ciBzZWxlY3QgZXhhbXBsZXMsIGVjaG8gPSBUUlVFfQooZGF0ZWRhdGEgPC0gc2VsZWN0KGZsaWdodHMsIHllYXIsIG1vbnRoLCBkYXkpKQooYnVuY2hvY29scyA8LSBzZWxlY3QoZmxpZ2h0cywgeWVhcjpkYXkpKQoobm9kYXRlcyA8LSBzZWxlY3QoZmxpZ2h0cywgLSh5ZWFyOmRheSkpKQpgYGAKCiMjIFBhcnRpYWwgbmFtZXMgIAoKYHNlbGVjdCgpYCBkb2VzIG5vdCBoYXZlIHRvIHVzZSBleGFjdCBjb2x1bW4gbWF0Y2hlcy4gIAoKWW91IGNhbiB1c2UgcGFydGlhbCBuYW1lcyBhbmQgcmVndWxhciBleHByZXNzaW9uczogIAoKKiBgc3RhcnRzX3dpdGgoImZvbyIpYCAgCiogYGVuZHNfd2l0aCgiYmFyIilgICAKKiBgY29udGFpbnMoImZvb2JhciIpYCAgCiogYG1hdGNoZXMoc29tZV9yZWdleClgICAKKiBgbnVtX3JhbmdlKCJ4IiwgMTozKWAgbWF0Y2hlcyB4MSwgeDIgYW5kIHgzICAKCiMjIFZhcmlhbnRzIG9uIGBzZWxlY3QoKWAKCllvdSBjYW4gdXNlIHNlbGVjdCgpIHRvIHJlbmFtZSBhbmQgcmUtb3JnYW5pemUgY29sdW1ucyB0byBzb21lIGV4dGVudC4gRm9yIGV4YW1wbGU6ICAKCiogYHJlbmFtZSgpYCBpcyBjb25zaWRlcmVkIGEgdmFyaWFudCBvZiBzZWxlY3QoKSB3aGVyZSB5b3UgdGFrZSBhIGNvbHVtbiwgY2hhbmdlIGl0cyBuYW1lLCBhbmQga2VlcCBhbGwgb3RoZXIgY29sdW1ucyBhcyB3ZWxsLiBJZiB5b3UgdXNlIGBzZWxlY3QoKWAgdG8gcmVuYW1lIGEgY29sdW1uIHlvdSB3aWxsIGxvc2UgYWxsIG90aGVyIGNvbHVtbnMuICAKKiBgZXZlcnl0aGluZygpYCBpcyBhIGhlbHBlciBmb3Igc2VsZWN0KCkgdGhhdCBsZXRzIHlvdSBtb3ZlIG9uZSBvciBhIGZldyBjb2x1bW5zIHRvIHRoZSBiZWdpbm5pbmcgKGxlZnQpIG9mIHRoZSB0YWJsZSwgd2hpbGUgcmV0YWluaW5nIGFsbCBvdGhlciBjb2x1bW5zLiAgCgpgYGB7ciBzZWxlY3QgdmFyaWFudCBleGFtcGxlc30KKGJldHRlcm5hbWVzIDwtIHJlbmFtZShmbGlnaHRzLCB0YWlsX251bSA9IHRhaWxudW0pCikKKGxvc3RteWNvbHMgPC0gc2VsZWN0KGZsaWdodHMsIHRhaWxfbnVtPXRhaWxudW0pKQoodHdlYWtjb2xzIDwtIHNlbGVjdChmbGlnaHRzLCB0aW1lX2hvdXIsIGFpcl90aW1lLCBldmVyeXRoaW5nKCkpKQoKYGBgCgoKIyMgRXhlcmNpc2VzIGZvciBzZWxlY3QoKSAgCjEuIEJyYWluc3Rvcm0gYXMgbWFueSB3YXlzIGFzIHBvc3NpYmxlIHRvIHNlbGVjdCBkZXBfdGltZSwgZGVwX2RlbGF5LCBhcnJfdGltZSwgYW5kIGFycl9kZWxheSBmcm9tIGZsaWdodHMuICAKCjIuIFdoYXQgaGFwcGVucyBpZiB5b3UgaW5jbHVkZSB0aGUgbmFtZSBvZiBhIHZhcmlhYmxlIG11bHRpcGxlIHRpbWVzIGluIGEgc2VsZWN0KCkgY2FsbD8gIAoKMy4gV2hhdCBkb2VzIHRoZSBhbnlfb2YoKSBmdW5jdGlvbiBkbz8gV2h5IG1pZ2h0IGl0IGJlIGhlbHBmdWwgaW4gY29uanVuY3Rpb24gd2l0aCB0aGlzIHZlY3Rvcj8gIAoKYHZhcnMgPC0gYygieWVhciIsICJtb250aCIsICJkYXkiLCAiZGVwX2RlbGF5IiwgImFycl9kZWxheSIpYCAgCgo0LiBEb2VzIHRoZSByZXN1bHQgb2YgcnVubmluZyB0aGUgZm9sbG93aW5nIGNvZGUgc3VycHJpc2UgeW91PyBIb3cgZG8gdGhlIHNlbGVjdCBoZWxwZXJzIGRlYWwgd2l0aCBjYXNlIGJ5IGRlZmF1bHQ/IEhvdyBjYW4geW91IGNoYW5nZSB0aGF0IGRlZmF1bHQ/ICAKCmBzZWxlY3QoZmxpZ2h0cywgY29udGFpbnMoIlRJTUUiKSlgICAKCmBgYHtyIHNlbGVjdCBleGVyY2lzZXN9CgpgYGAKCg==